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What Are Naturally Occurring School Lotteries 
and How Do We Identify Them? 


By Rebecca Unterman 


This post is one ina series highlighting MDRC’s methodological work. Contributors discuss the refine- 
ment and practical use of research methods being employed across our organization. 


Researchers across disciplines have long taken advantage of natural experiments to study the ef- 
fects of policies at full scale (one of the first and best-known examples of this being researchers’ use 
of the Vietnam draft lottery to identify the effect of serving in the military on long-term earnings). 
In the past decade rapid growth in the number of charter schools and school district choice systems 
has provided education researchers with exciting opportunities to do the same: to use naturally 
occurring pockets of randomization to rigorously study the effects of policy-relevant education 
reforms that are already in place, often on a large scale. What’s more, this work can often be done 
quickly and relatively cheaply, relying solely on preexisting administrative records. Finally, when 
done retrospectively, the design allows for long-term follow-up of student outcomes within a short 
time frame. 


WHAT IS A NATURALLY OCCURRING SCHOOL LOTTERY? 


When a lottery is used to decide which students receive the opportunity to attend a school, it can 
create the conditions for a natural experiment. Because students’ school assignment is random, 
there should be no measurable or unmeasurable differences between the students who win and 
those who lose the opportunity to attend the school; therefore, any differences in the students’ 
future outcomes can be attributed to the opportunity to attend the school. In other words, re- 
searchers can identify the causal effect of the school model on students. Naturally occurring lot- 
teries most often take place within relatively simple school-managed processes or within centrally 
managed admissions processes. 


SCHOOL-MANAGED ADMISSIONS 


The simplest version of a school lottery occurs when a school independently manages its own as- 
signment process and students (or their parents) submit applications to the school. Based on its 
mission and any state or federal funding rules, the school may set priorities for groups (such as 
siblings, or students from a certain geographic area). If more students apply than there are seats 
available, within its priority groups, the school uses a lottery to randomly choose which students 
will be offered a seat. MDRC’s study of the SEED charter school in Washington, DC, followed this 
approach.? 


To use a school-managed lottery, researchers must assess whether the process was truly random 
and the data are sufficient to be used for an evaluation. In our SEED study we attended the public 
lottery and observed the process; we also walked through the lottery documentation with school 
leaders and traced students through the assignment process into enrollment the following fall (and 
beyond). Ultimately, we tracked the results student by student and verified that at every point the 
students’ lottery results determined whether they were admitted. 


+Other examples are the nationwide study of KIPP Charter Schools, a study of New York City’s Harlem Children’s Zone mod- 
el, and the impact of Early College High Schools. 


It is critical to know how the waiting list is handled by school personnel to determine whether 
wait-listed students later admitted should be classified as treatment or control group members. If 
students are placed on the waiting list in random order as a result of the lottery, their admissions 
offers may be seen as an extension of the lottery process. But if those who lose the lottery must 
take additional steps to be placed on the waiting list, then wait-list admissions offers may be affect- 
ed by other student or family characteristics and are no longer randomly determined. In that case, 
the initial lottery results should be used to determine students’ treatment classification, and any 
post-lottery movement (even via the waiting list) needs to be accounted for in the analysis. Final- 
ly, in areas saturated with school options, students may participate in multiple independently run 
lotteries and may be selected for admission in more than one school. If all data are available, this 
situation can be addressed analytically. (These analytic issues will be addressed in a future post.) 


CENTRALLY MANAGED ADMISSIONS 


Alternatively, a school district may centrally manage all school admissions, and students apply to 
one or more schools through a district-run process. The district then uses an algorithm that takes 
student and school preferences into account to make admission decisions across multiple schools. 
In the case of a nonselective school, the district would not exercise preferences over which stu- 
dents to admit and would rely on a random process. In effect, the algorithm assigns students a 
random number and uses this to break ties when more students want to go to a school than the 
school can serve, creating the equivalent of a lottery. 


MDRC has built a portfolio of work taking advantage of such randomization to study programs on 
a large scale. In 2010 Howard Bloom, Saskia Levy Thompson, and | wrote our first report on the 
Small Schools of Choice project, which used lotteries within the New York City school assignment 
process to study the effects of a large-scale education reform initiative: a set of about 100 new 
small high schools created to serve educationally and economically disadvantaged students (Bloom 
and Unterman, 2014). For Small Schools of Choice, a lottery occurs when the number of eligible 
students living within the geographic area who have attended a high school open house and want 
to attend the school is more than the school can serve; eligible students with lower priorities will 
not be admitted. (MDRC teams have identified similar lotteries for Boston Public Schools prekinder- 
garten programs and NYC career and technical education programs.) 


For work involving centrally managed admissions, a fundamental step is to engage with the district 
and listen to the staff to gain a clear understanding of the assignment rules. Afterward, researchers 
must carefully check that understanding with patterns in the assignment data. Once they are cer- 
tain they understand the process, researchers can deduce where pockets of randomization have 
occurred. Thus, the practice of identifying naturally occurring lotteries involves both a theoretical 
understanding of the assignment process and an empirical review of the data. A final check of the 
lottery’s internal validity is done by comparing the baseline characteristics of students who won and 
who lost the opportunity to attend the school, to ensure they do not differ systematically. 


Once such lotteries are identified, the analysis proceeds as it would in any other multisite ran- 
domized controlled trial. However, the lottery identification process (and resulting sample) bring 
to the foreground a few methodological issues: Statistical power may be weaker than anticipated 
because of many small lotteries or because of skewed treatment-control group ratios (for example, 
if a school is very popular, a large number of students may compete for a small number of seats). 
Compliance may be imperfect if many students who win admission do not attend a school and stu- 
dents who do not win admission attend anyway (via subsequent steps in the assignment process). 
And if the oversubscribed priority group does not represent the majority of students attending the 
school or the sample selection process after school assignment is opaque, the generalizability of the 
findings may be constrained. These issues will be the subject of a future post. 


